Is a Website or E-Learning Product Working Well? How Many Users Should You Test?

How can one determine efficiently if an informational website or an e-learning product is working well? Relatively small numbers of the target audience are needed to improve a product during formative evaluation and usability testing as part of product development and revision cycles. However, during summative evaluation, how many subjects are needed to determine product effectiveness? When investigating the number of subjects needed for usability tests, a Poisson probability model was found to be a reasonable fit to extant data (Nielsen & Landauer, 1993; Virzi, 1990, 1992). However, this model was chosen on the basis of the number of subjects needed to identify important usability problems with a product, not for determining its effectiveness. To determine if a Website or e-learning product is working well, we investigated the predictive validity of a discrete Bayesian decision model: the Sequential Probability Ratio Test (SPRT) -originally developed by Wald (1947). Fifty-one people representing a campus community participated in a usability test of the university library online catalog search tool, and the results were analyzed post hoc with SPRT re-enactments to simulate sequential decision making after testing each subject. Across a range of parameters, the Bayesian SPRT reached the same conclusion as reflected by the entire sample with many fewer subjects, utilizing typically small Type I and II error rates. The study provides evidence of the usefulness of the SPRT decision model in situations where determination of effectiveness is the goal (product works well or not). The SPRT maximizes efficiency by testing only as many users as necessary to reach a confident conclusion.